Aspects of the present invention relate in general to process management, and more particularly, to managing the processing of requests in a manner that provides high throughput without neglecting the processing of some processing requests.
A service oriented architecture is one in which a processing entity may outsource processing requests to a number of remote processing systems. These remote processing servers may further outsource a processing request to other remote processing systems. Service oriented architecture may organize applications as a hierarchy or even as a web of interacting services.
To provide efficient throughput of the system, a computing system which is processing requests should be able to continue to respond to its clients while waiting for responses from various servers over which it has little or no control. To do so, computing systems are often configured with the capability to process multiple requests in parallel. Thus, when some requests are held up for various reasons, there may be other requests which are able to be processed. The aim of such an approach is to typically maximize throughput in the service provider and not necessarily to minimize the response time of the server systems to which processing requests are sent.
However, this approach presents some challenges. For example, if the number of requests that are currently in progress becomes too large, the effort required to manage them can become significant and might actually degrade performance. Furthermore, there is also the possibility that the functions which decide the order in which processing requests are processed may neglect a particular request in favor of others. This may lead to unnecessarily long delays. For example, if the computing system is trying to work on several requests in parallel, it is possible that one request will be neglected while other requests receive a disproportionately large share of the available processing time.